from sklearn_benchmarks.reporting.hp_match import HPMatchReporting

Benchmarking prediction: scikit-learn vs. ONNX Runtime

reporting = HPMatchReporting("onnx", config="config.yml")
reporting.run()

KNeighborsClassifier_brute_force

onnx (1.9.0) vs. scikit-learn (1.0.dev0)

All estimators share the following hyperparameters: algorithm=brute.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration	std_duration	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	predict	100000	1000	100	16.873	0.004	0.0	0.017	-1	1	0.674	1.788	0.048	9.436	0.252	See	See
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	16.873	0.004	0.0	0.017	-1	1	0.674	0.023	0.002	741.521	52.705	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.309	0.013	0.0	0.309	-1	1	1.000	1.788	0.048	0.173	0.009	See	See
3	KNeighborsClassifier_brute_force	predict	100000	1	100	0.309	0.013	0.0	0.309	-1	1	1.000	0.023	0.002	13.585	1.118	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	16.899	0.014	0.0	0.017	1	100	0.845	1.855	0.003	9.109	0.015	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1000	100	16.899	0.014	0.0	0.017	1	100	0.845	0.020	0.000	850.551	5.649	See	See
6	KNeighborsClassifier_brute_force	predict	100000	1	100	0.307	0.007	0.0	0.307	1	100	1.000	1.855	0.003	0.165	0.004	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1	100	0.307	0.007	0.0	0.307	1	100	1.000	0.020	0.000	15.445	0.386	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1000	100	16.884	0.076	0.0	0.017	-1	100	0.845	2.497	0.036	6.762	0.101	See	See
9	KNeighborsClassifier_brute_force	predict	100000	1000	100	16.884	0.076	0.0	0.017	-1	100	0.845	0.023	0.002	744.091	54.803	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1	100	0.304	0.007	0.0	0.304	-1	100	1.000	2.497	0.036	0.122	0.003	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.304	0.007	0.0	0.304	-1	100	1.000	0.023	0.002	13.412	1.040	See	See
12	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.294	0.131	0.0	0.017	-1	5	0.729	2.476	0.041	6.984	0.127	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.294	0.131	0.0	0.017	-1	5	0.729	0.025	0.005	683.829	135.999	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.306	0.013	0.0	0.306	-1	5	1.000	2.476	0.041	0.124	0.006	See	See
15	KNeighborsClassifier_brute_force	predict	100000	1	100	0.306	0.013	0.0	0.306	-1	5	1.000	0.025	0.005	12.111	2.460	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	16.834	0.027	0.0	0.017	1	1	0.674	1.084	0.005	15.529	0.080	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1000	100	16.834	0.027	0.0	0.017	1	1	0.674	0.019	0.000	908.833	8.890	See	See
18	KNeighborsClassifier_brute_force	predict	100000	1	100	0.311	0.011	0.0	0.311	1	1	1.000	1.084	0.005	0.287	0.010	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1	100	0.311	0.011	0.0	0.311	1	1	1.000	0.019	0.000	16.800	0.620	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1000	100	16.947	0.124	0.0	0.017	1	5	0.729	1.842	0.006	9.202	0.074	See	See
21	KNeighborsClassifier_brute_force	predict	100000	1000	100	16.947	0.124	0.0	0.017	1	5	0.729	0.019	0.000	878.641	9.730	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1	100	0.307	0.006	0.0	0.307	1	5	1.000	1.842	0.006	0.167	0.003	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	100	0.307	0.006	0.0	0.307	1	5	1.000	0.019	0.000	15.916	0.344	See	See
24	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.543	0.010	0.0	0.004	-1	1	0.856	1.596	0.022	2.220	0.031	See	See
25	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.543	0.010	0.0	0.004	-1	1	0.856	0.008	0.002	468.856	124.892	See	See
26	KNeighborsClassifier_brute_force	predict	100000	1	2	0.245	0.007	0.0	0.245	-1	1	1.000	1.596	0.022	0.153	0.005	See	See
27	KNeighborsClassifier_brute_force	predict	100000	1	2	0.245	0.007	0.0	0.245	-1	1	1.000	0.008	0.002	32.394	8.677	See	See
28	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.594	0.010	0.0	0.004	1	100	0.908	2.014	0.002	1.784	0.005	See	See
29	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.594	0.010	0.0	0.004	1	100	0.908	0.003	0.000	1411.274	101.637	See	See
30	KNeighborsClassifier_brute_force	predict	100000	1	2	0.247	0.006	0.0	0.247	1	100	1.000	2.014	0.002	0.123	0.003	See	See
31	KNeighborsClassifier_brute_force	predict	100000	1	2	0.247	0.006	0.0	0.247	1	100	1.000	0.003	0.000	97.041	7.320	See	See
32	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.613	0.017	0.0	0.004	-1	100	0.908	2.628	0.023	1.375	0.013	See	See
33	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.613	0.017	0.0	0.004	-1	100	0.908	0.009	0.002	418.378	76.412	See	See
34	KNeighborsClassifier_brute_force	predict	100000	1	2	0.244	0.005	0.0	0.244	-1	100	1.000	2.628	0.023	0.093	0.002	See	See
35	KNeighborsClassifier_brute_force	predict	100000	1	2	0.244	0.005	0.0	0.244	-1	100	1.000	0.009	0.002	28.297	5.203	See	See
36	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.548	0.017	0.0	0.004	-1	5	0.890	2.695	0.072	1.316	0.036	See	See
37	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.548	0.017	0.0	0.004	-1	5	0.890	0.010	0.004	341.099	119.731	See	See
38	KNeighborsClassifier_brute_force	predict	100000	1	2	0.244	0.007	0.0	0.244	-1	5	1.000	2.695	0.072	0.090	0.004	See	See
39	KNeighborsClassifier_brute_force	predict	100000	1	2	0.244	0.007	0.0	0.244	-1	5	1.000	0.010	0.004	23.428	8.253	See	See
40	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.563	0.021	0.0	0.004	1	1	0.856	0.992	0.005	3.592	0.027	See	See
41	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.563	0.021	0.0	0.004	1	1	0.856	0.002	0.000	2006.210	164.568	See	See
42	KNeighborsClassifier_brute_force	predict	100000	1	2	0.245	0.008	0.0	0.245	1	1	1.000	0.992	0.005	0.247	0.008	See	See
43	KNeighborsClassifier_brute_force	predict	100000	1	2	0.245	0.008	0.0	0.245	1	1	1.000	0.002	0.000	137.886	12.065	See	See
44	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.550	0.014	0.0	0.004	1	5	0.890	2.014	0.005	1.762	0.008	See	See
45	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.550	0.014	0.0	0.004	1	5	0.890	0.003	0.000	1383.778	84.280	See	See
46	KNeighborsClassifier_brute_force	predict	100000	1	2	0.250	0.007	0.0	0.250	1	5	1.000	2.014	0.005	0.124	0.004	See	See
47	KNeighborsClassifier_brute_force	predict	100000	1	2	0.250	0.007	0.0	0.250	1	5	1.000	0.003	0.000	97.343	6.555	See	See

KNeighborsClassifier_kd_tree

onnx (1.9.0) vs. scikit-learn (1.0.dev0)

All estimators share the following hyperparameters: algorithm=kd_tree.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration	std_duration	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	113.710	0.000	0.0	0.114	-1	5	0.980	0.896	0.017	126.894	2.372	See	See
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	113.710	0.000	0.0	0.114	-1	5	0.980	0.003	0.000	43891.254	6013.315	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.650	0.137	0.0	2.650	-1	5	1.000	0.896	0.017	2.957	0.163	See	See
3	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.650	0.137	0.0	2.650	-1	5	1.000	0.003	0.000	1022.732	149.772	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	109.197	0.000	0.0	0.109	-1	100	0.981	2.891	0.028	37.768	0.365	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	109.197	0.000	0.0	0.109	-1	100	0.981	0.004	0.001	28463.917	6528.721	See	See
6	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.596	0.075	0.0	2.596	-1	100	1.000	2.891	0.028	0.898	0.027	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.596	0.075	0.0	2.596	-1	100	1.000	0.004	0.001	676.693	156.435	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	116.010	0.000	0.0	0.116	-1	1	0.966	0.458	0.004	253.433	2.039	See	See
9	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	116.010	0.000	0.0	0.116	-1	1	0.966	0.002	0.000	48331.778	7664.319	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.634	0.118	0.0	2.634	-1	1	1.000	0.458	0.004	5.754	0.263	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.634	0.118	0.0	2.634	-1	1	1.000	0.002	0.000	1097.291	180.864	See	See
12	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	120.113	0.000	0.0	0.120	1	100	0.981	5.180	0.048	23.187	0.215	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	120.113	0.000	0.0	0.120	1	100	0.981	0.003	0.000	46571.393	8830.541	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.699	0.155	0.0	2.699	1	100	1.000	5.180	0.048	0.521	0.030	See	See
15	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.699	0.155	0.0	2.699	1	100	1.000	0.003	0.000	1046.435	207.307	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	123.432	0.000	0.0	0.123	1	5	0.980	1.493	0.012	82.651	0.640	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	123.432	0.000	0.0	0.123	1	5	0.980	0.001	0.000	117185.724	30503.905	See	See
18	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.617	0.091	0.0	2.617	1	5	1.000	1.493	0.012	1.752	0.062	See	See
19	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.617	0.091	0.0	2.617	1	5	1.000	0.001	0.000	2484.263	652.357	See	See
20	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	119.584	0.000	0.0	0.120	1	1	0.966	0.777	0.006	153.875	1.280	See	See
21	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	119.584	0.000	0.0	0.120	1	1	0.966	0.001	0.000	135523.839	32295.014	See	See
22	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.611	0.089	0.0	2.611	1	1	1.000	0.777	0.006	3.360	0.118	See	See
23	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.611	0.089	0.0	2.611	1	1	1.000	0.001	0.000	2958.955	712.364	See	See
24	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.038	0.001	0.0	0.000	-1	5	0.898	0.023	0.001	1.644	0.088	See	See
25	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.038	0.001	0.0	0.000	-1	5	0.898	0.002	0.000	16.946	2.007	See	See
26	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	-1	5	1.000	0.023	0.001	0.224	0.013	See	See
27	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	-1	5	1.000	0.002	0.000	2.312	0.277	See	See
28	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.062	0.003	0.0	0.000	-1	100	0.894	0.035	0.002	1.768	0.113	See	See
29	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.062	0.003	0.0	0.000	-1	100	0.894	0.002	0.000	27.532	5.645	See	See
30	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	-1	100	1.000	0.035	0.002	0.149	0.008	See	See
31	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	-1	100	1.000	0.002	0.000	2.314	0.466	See	See
32	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.037	0.001	0.0	0.000	-1	1	0.885	0.022	0.002	1.716	0.136	See	See
33	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.037	0.001	0.0	0.000	-1	1	0.885	0.002	0.000	17.874	3.111	See	See
34	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	-1	1	1.000	0.022	0.002	0.237	0.019	See	See
35	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	-1	1	1.000	0.002	0.000	2.467	0.432	See	See
36	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.061	0.000	0.0	0.000	1	100	0.894	0.033	0.000	1.822	0.016	See	See
37	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.061	0.000	0.0	0.000	1	100	0.894	0.001	0.000	100.286	21.554	See	See
38	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	1	100	1.000	0.033	0.000	0.157	0.007	See	See
39	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	1	100	1.000	0.001	0.000	8.652	1.901	See	See
40	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.038	0.000	0.0	0.000	1	5	0.898	0.020	0.000	1.879	0.019	See	See
41	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.038	0.000	0.0	0.000	1	5	0.898	0.001	0.000	63.972	14.223	See	See
42	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	1	5	1.000	0.020	0.000	0.258	0.007	See	See
43	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	1	5	1.000	0.001	0.000	8.787	1.966	See	See
44	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.038	0.002	0.0	0.000	1	1	0.885	0.019	0.000	2.007	0.089	See	See
45	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.038	0.002	0.0	0.000	1	1	0.885	0.001	0.000	63.187	13.488	See	See
46	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	1	1	1.000	0.019	0.000	0.276	0.008	See	See
47	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	1	1	1.000	0.001	0.000	8.679	1.829	See	See

HistGradientBoostingClassifier_best

onnx (1.9.0) vs. scikit-learn (1.0.dev0)

All estimators share the following hyperparameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

	estimator	function	n_samples_train	n_samples	n_features	n_iter	mean_duration	std_duration	iteration_throughput	latency	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	predict	100000	1000	100	300	0.45	0.017	0.002	0.0	0.818	0.106	0.002	4.238	0.175	See	See